Findings

1. Blocking: publication_name is accepted as arbitrary non-empty text in FE, but both the create/update path and the cleanup path splice it directly into SQL (fs_brokers/cdc_client/src/main/java/io/debezium/connector/postgresql/connection/PostgresReplicationConnection.java, fs_brokers/cdc_client/src/main/java/org/apache/doris/cdcclient/source/reader/postgres/PostgresSourceReader.java). A user-provided name such as MyPub or pub-name is valid only when quoted in PostgreSQL, so this feature currently accepts values that later fail at runtime.
2. Blocking: auto-created-resource cleanup is detected only by prefix (doris_pub_ / doris_cdc_) in fs_brokers/cdc_client/src/main/java/org/apache/doris/cdcclient/source/reader/postgres/PostgresSourceReader.java. That violates the PR contract that user-provided names are preserved: if a user explicitly sets publication_name = doris_pub_shared or slot_name = doris_cdc_shared, DROP JOB will still delete that external resource.

Critical Checkpoints

• Goal of current task: Partially met. Happy-path per-table publication / SHOW visibility are implemented and covered, but the two blockers above break custom-name support and cleanup semantics.
• Modification size / focus: Yes, mostly focused.
• Concurrency: No new concurrency-sensitive path identified.
• Lifecycle management: Needs work. External PostgreSQL resource ownership is inferred from name prefixes instead of persisted ownership.
• Configuration items: New source properties were added, but validation is incomplete for PostgreSQL identifier rules and cleanup semantics.
• Incompatible changes: No FE/BE protocol or storage compatibility issue found.
• Parallel code paths: No matching non-PostgreSQL path appears to need the same publication logic.
• Special conditional checks: Prefix-based auto-created detection is not a safe condition.
• Test coverage: Positive regression and unit coverage exists; missing negative tests for custom identifiers and user-provided prefixed names.
• Observability: Sufficient for this change.
• Transaction / persistence: Source property persistence and replay path looks okay.
• Data writes / modifications: Unsafe external-resource deletion behavior remains.
• FE-BE variable passing: Not applicable.
• Performance: No major concern found.
• Other issues: None beyond the blockers above.

Pull request overview

Updates Doris PostgreSQL CDC streaming jobs to use per-table PostgreSQL publications (filtered mode) and to persist/show slot/publication names via job source properties, improving WAL decoding efficiency and operational visibility.

Changes:

• Add slot_name / publication_name source properties (auto-generated by FE when omitted) and surface them in SHOW JOBS output.
• Switch Debezium publication autocreation to filtered mode and add publish_via_partition_root=true for PG 13+ when creating filtered publications.
• Add regression + unit tests covering generated names, per-table publication contents, and cleanup on job drop.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 6 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

I found blocking compatibility and resource lifecycle regressions in the PostgreSQL CDC changes.

1. fs_brokers/cdc_client/src/main/java/org/apache/doris/cdcclient/source/reader/postgres/PostgresSourceReader.java: legacy PostgreSQL CDC jobs created before this PR are no longer restart compatible. Before this change the reader never set publication.name, so those jobs ran against the Debezium default dbz_publication with ALL_TABLES. This patch now falls back to doris_pub_<jobId>, classifies the job as user managed, sets publication.autocreate.mode=disabled, and validates that the new publication exists. Replayed jobs whose persisted sourceProperties do not contain publication_name will fail initialization after upgrade, and their legacy doris_cdc_<jobId> slot will also never be cleaned up on drop.
2. fs_brokers/cdc_client/src/main/java/org/apache/doris/cdcclient/source/reader/postgres/PostgresSourceReader.java: cleanup ownership is inferred from the current slot and publication names. Because these source properties remain mutable through ALTER JOB, changing either name after creation makes close() skip cleanup and leaks the original Doris created slot or publication. Ownership needs to be persisted at creation time, or these properties need to be immutable.

Critical checkpoint conclusions:

• Goal: the PR aims to add per table PostgreSQL publications, custom slot and publication names, SHOW visibility, and cleanup. The new job happy path is covered, but the current code does not safely preserve existing jobs or resource lifecycle.
• Minimality and focus: the change stays localized, but it couples new naming, ownership inference, and cleanup without persisting compatibility or ownership state.
• Concurrency: no new lock order or heavy under lock issue stood out; the reader still uses the existing slot creation lock.
• Lifecycle: slot and publication lifecycle tracking is not reliable because it depends on current name patterns rather than persisted ownership or legacy metadata.
• Configuration items: new source properties are validated, but compatibility for already persisted jobs is incomplete.
• Incompatible changes: yes. Publication naming and default behavior changed without compatibility handling for pre existing persisted jobs.
• Parallel paths: replay or upgrade and ALTER JOB paths are not updated consistently with the create path.
• Special conditions: isAutoGenerated() is too strong as an ownership predicate; it misses legacy Doris managed resources and any auto resource after name mutation.
• Test coverage: validator unit tests and one regression test cover only the fresh job happy path; there is no replay or upgrade or ALTER and cleanup coverage for the new semantics.
• Observability: validation errors are actionable, but the cleanup path still logs only the message on failure, which leaves less detail for diagnosis.
• Transaction and persistence: no FE journal format change is needed, but persisted sourceProperties from older jobs are not interpreted compatibly.
• Data modification or external state: PostgreSQL publication and slot lifecycle can be broken or leaked.
• FE BE variable passing: not applicable.
• Performance: no material hot path issue stood out in the touched code.

Findings

1. Blocking: slot_name is now treated as independently user-owned, but the implementation never proves that a user-provided slot actually exists. initialize() skips the explicit createSlotForGlobalStreamSplit() branch when slot_name is set, yet the downstream replication path still calls PostgresReplicationConnection.initReplicationSlot() and auto-creates the slot whenever getSlotInfo() returns INVALID. In practice, CREATE JOB ... "slot_name" = "my_slot" will silently create my_slot if the caller forgot to pre-create it, and close() will then preserve that Doris-created slot forever because ownership is inferred from the config flag. This breaks the new ownership contract and leaks replication slots.
2. Blocking: the new validation for existing publications only checks name existence plus table membership (pg_publication_tables), but partition-table support in this PR also depends on publish_via_partition_root = true. A user-supplied or legacy publication for a partitioned root table can satisfy the current validation and still publish changes with the leaf partition identities instead of the root identity, while Doris still builds table.include.list from the configured root names. That causes partition changes to be missed even though initialization succeeds. The validation needs to check pg_publication.pubviaroot (and for Doris-owned publications, ensure the created publication keeps that setting).

Critical Checkpoints

• Goal of current task: Partially met. The PR adds filtered publications, SHOW visibility, custom names, cleanup, and tests for several happy paths, but the two cases above still violate the new PostgreSQL resource-ownership and partition-table correctness goals.
• Modification size / focus: Yes. The change set stays focused on PostgreSQL CDC job configuration, runtime initialization, cleanup, and tests.
• Concurrency: No blocking concurrency issue found. The existing slot-creation lock is preserved, and no new lock-order risk stood out in FE.
• Lifecycle management: Not fully correct. External PostgreSQL resource ownership is now modeled per resource, but user-provided slot lifecycle is not actually enforced because missing slots are still auto-created later.
• Configuration items added: Yes. slot_name and publication_name are validated and surfaced in SHOW, but the runtime behavior is still inconsistent with the intended ownership semantics in the slot path.
• Incompatible changes: No FE/BE protocol or storage compatibility issue found in the final patch set.
• Functionally parallel code paths: The relevant PostgreSQL create/alter/show/cleanup paths were updated consistently; no matching MySQL path needs the same publication logic.
• Special conditional checks: The new ownership and validation branches are conceptually reasonable, but the publication check is incomplete because it ignores pubviaroot for partition roots.
• Test coverage: Positive coverage is improved with FE unit tests and a new regression suite, but there is still no negative coverage for the missing-slot user-owned case or for partitioned tables with user-supplied publications lacking publish_via_partition_root.
• Observability: Adequate for review purposes. Error messages are generally actionable.
• Transaction / persistence modifications: No EditLog / replay issue found in the touched code paths.
• Data writes / modifications: Yes, this change touches external PostgreSQL publication and replication-slot state; the two issues above can lead to leaked external resources or silently missed partition change events.
• FE-BE variable passing: Not applicable here.
• Performance: No major hot-path performance regression found in the reviewed changes.
• Other issues: None beyond the two blockers above.

I found 2 blocking issues.

1. Existing PostgreSQL streaming jobs are not restart/rolling-upgrade compatible. Jobs created before this PR persisted no publication_name; they previously used Debezium's default dbz_publication, but the new fallback resolves missing values to doris_pub_{jobId}. After restart or upgrade the same job will use a different publication, may need new CREATE/ownership privileges, and can leave the original publication orphaned. SHOW JOBS also reports the synthesized new name instead of the historical one.
2. PostgresResourceValidator.validate() opens a JdbcClient and only closes the JDBC Connection. The backing Hikari datasource is never closed, so each CREATE JOB / PG TVF validation leaks FE-side resources.

Checkpoint conclusions:

• Goal: The PR improves new-job slot/publication ownership and cleanup, but it is not yet correct for existing PG jobs across restart/upgrade.
• Minimality: The change is reasonably focused, but the fallback publication default changes persisted-job behavior more broadly than intended.
• Concurrency: I did not find a new locking/deadlock issue; the new validation path appears to run outside FE job-manager locks.
• Lifecycle: Not correct in the validator path because the JdbcClient is never closed.
• Config items: slot_name / publication_name are added and CREATE-time immutability is enforced.
• Compatibility: Broken for pre-existing PG jobs whose persisted config lacks publication_name.
• Parallel paths: The new validator is wired into both CREATE JOB and TVF paths; the missing path is upgrade/restart coverage for old jobs.
• Conditional checks: The ALTER immutability checks look reasonable and are documented.
• Test coverage: Good new regression coverage for new-job ownership/immutability, but missing upgrade/restart coverage and validator lifecycle coverage.
• Observability: Existing logs look sufficient; I do not see a required new metric here.
• Transaction/persistence: No new EditLog format issue found, but backward-compat handling for old persisted jobs is missing.
• Data/resource cleanup: Cleanup behavior looks correct for newly created jobs; upgraded old jobs can still leak the original publication.
• FE/BE variable passing: The new properties propagate on the touched create/TVF paths.
• Performance: No obvious hot-path regression beyond the leaked validator datasource.
• Other: None.

Findings:

1. Legacy cdc_stream PostgreSQL streaming jobs are not upgrade/restart compatible. Replayed jobs whose persisted SQL never contained publication_name now materialize doris_pub_{jobId} in the TVF path, so they reconnect with a different publication than the pre-PR dbz_publication behavior.
2. Standalone cdc_stream with a user-supplied missing slot_name still auto-creates that slot inside the cdc client, but close() now treats it as user-owned and preserves it, leaking replication slots on ad hoc queries.
3. The TVF-backed CREATE JOB ... DO INSERT ... SELECT FROM cdc_stream(...) path still skips the new FE-side slot_name / publication_name identifier validation, so invalid identifiers are accepted at create time and only fail later inside the cdc client.
4. User-managed publications for partitioned root tables are accepted without checking pg_publication.pubviaroot, so partition changes can be silently skipped even though validation succeeds.

Checkpoint conclusions:

• Goal: Partially achieved for newly created jobs, but the current code does not fully satisfy the PR goal because the four blocking cases above still break correctness/compatibility.
• Minimality: The change set is focused, but the new default-name injection expands behavior in the legacy TVF replay path more than intended.
• Concurrency: No new FE lock-order or deadlock issue found; the new logic appears outside catalog-lock critical sections, and SLOT_CREATION_LOCK remains localized.
• Special lifecycle / replay: Not correct yet. Legacy TVF jobs and standalone TVF queries now take different ownership paths across create/replay/close, so lifecycle handling is inconsistent.
• Configuration items: slot_name / publication_name are added and ALTER immutability is enforced in the touched job paths, but create-time validation is still inconsistent between FROM-TO jobs and TVF-backed jobs.
• Compatibility / incompatible changes: Broken for existing TVF-backed PostgreSQL jobs whose persisted SQL lacks publication_name; after upgrade/restart they reconnect with a different publication.
• Parallel code paths: FROM-TO and TVF-backed create paths are not fully aligned; only the FROM-TO path gets DataSourceConfigValidator.validateSource(), and only Doris-managed publications get the partition-root invariant.
• Conditional checks: The publication coverage check is insufficient for partition roots; table-name coverage alone does not prove correctness.
• Test coverage: Good new coverage for new-job ownership/cleanup, but missing legacy TVF upgrade/restart coverage, invalid-identifier coverage for TVF-backed CREATE JOB, standalone missing-slot leakage coverage, and user-managed partition-publication coverage.
• Observability: Existing logs look sufficient; I do not see a required new metric.
• Transaction / persistence: No new EditLog-format issue found, but persisted executeSql replay for old TVF jobs lacks backward-compat handling.
• Data writes / modifications: The missing pubviaroot validation can silently drop change events from partitioned tables, so data correctness is not fully guaranteed.
• FE/BE variable passing: The new properties propagate through several paths, but validation and replay semantics are still inconsistent across those paths.
• Performance: No material hot-path regression found beyond the correctness problems above.
• Other issues: None beyond the four blocking items above.